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1.0  INTRODUCTION 

Due  to  the  rising  costs  of  today’s  weapon  systems,  the 
U.S.  Department  of  Defense  (DOD)  continues  to 
implement  strategies  to  reform  its  acquisition  and 
procurement  process.  One  such  strategy  seeks  to 
reduce  the  cost  of  developing  systems  by  purchasing 
commercial  off-the-shelf  (COTS)  technology.  The 
COTS  technology  ranges  from  components  used  to 
build  a particular  weapon  system  to  functional  pieces 
of  gear  used  to  support  the  weapon  system,  i.e.,  support 
equipment.  The  COTS  technology  may  be  instituted  at 
the  inception  of  the  weapon  system  design  or  it  may  be 
inserted  into  the  support  of  the  weapon  system  at  any 
point  during  its  life  cycle.  The  COTS  technology  is 
intended  to  reduce  weapon  system  life-cycle  costs  by 
minimizing  the  expense  of  system  design  and  testing. 

While  using  COTS  technology  is  beneficial  to  the 
DOD,  several  factors  must  be  weighed  before  such 
technologies  can  be  introduced  effectively.  Above  all, 
the  typical  systems  engineering  thought  process  must 
be  adjusted  to  incorporate  the  potential  risks  of  COTS 
technology.  One  of  the  most  significant  risks  involves 
parts  obsolescence.  Systems  engineers  must  decide 
how  and  when  to  use  rapidly  changing  COTS 
technology  to  keep  pace  with  the  commercial 
technology  market.  Technology  manufacturers 
regularly  develop  new  versions  of  electronics  and 
software  and  new  designs  of  mechanical  parts.  These 
rapid  changes  lead  to  technology  “outpacing”  fielded 
military  systems,  which  often  have  long  life  spans  and 
require  legacy  parts  support.  Previously,  as  one  of  the 
most  influential  players  in  the  development  of 
technologies  such  as  electronics,  the  DOD  often 
“drove”  technology  development  to  fulfill  its  needs. 
Now,  increasing  demands  for  electronic  technologies 
from  all  sectors  of  the  market  (e.g.,  industrial, 
professional,  personal,  and  government)  have  lessened 
the  DOD’s  influence  on  the  pace  of  technology 
development.  And,  while  the  DOD’s  desire  to  field 
new  and  innovative  technologies  has  increased. 


acquisition  budgets  have  actually  decreased.  As  a 
result,  the  DOD  finds  it  more  difficult  to  drive  major 
price  efficiencies  than  in  the  past.  Because  the 
commercial  industry  currently  views  the  DOD  as  a 
different  kind  of  player  in  the  technology  market — one 
with  more  stringent  requirements  than  other 
customers — the  DOD  no  longer  can  easily  influence 
technology  suppliers  to  design,  test,  and  support  their 
products  in  the  manner  prescribed  by  the  DOD.  In 
short,  technology  suppliers  are  less  willing  to  guarantee 
the  configuration  design  stability  and  logistics  support 
required  by  DOD  systems  engineers  to  ensure  that  a 
weapon  system  will  be  adequately  supported 
throughout  its  life  cycle.  This  diminished  technology 
support,  if  not  managed  properly,  can  lead  to  parts 
obsolescence,  which  in  turn  can  lead  to  increased  life- 
cycle  costs  for  a weapon  system,  as  well  as  diminished 
mission  readiness. 

For  example,  there  is  an  inherent  risk  if  the  DOD 
procures  COTS  equipment  for  a specific  weapon 
system  and  the  technology  manufacturer  ceases  to 
provide  replacement  parts  because  technology  has 
advanced  since  that  equipment  was  fielded.  In  the  best- 
case  scenario,  the  manufacturer  designed  its  equipment 
using  open  architecture  and  either  the  new,  updated 
technology  parts  can  directly  replace  the  old  parts  in 
the  fielded  equipment  or  they  can  be  integrated  using  a 
manufacturer-supplied  interface.  In  the  worst-case 
scenario,  replacement  parts  are  not  available  because 
the  manufacturer  has  either  gone  out  of  business  or  did 
not  plan  to  supply  upgraded  or  original  parts  to  the 
DOD  over  the  lifetime  of  the  weapon  system.  In  either 
case,  the  DOD  will  have  to  cover  the  risk  to  mission 
readiness,  as  well  as  the  cost  of  replacing  obsolete  parts 
or  even  redesigning/modifying  the  equipment  to  make 
it  compatible  with  the  new  technology  parts. 

Additionally,  one  of  the  prevailing  and  flawed  opinions 
in  applying  COTS  technology  to  DOD  weapon  systems 
is  that  “if  the  technology  exists  in  the  commercial 
marketplace,  it  already  must  be  appropriate  for  use  in 
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the  military  and,  therefore,  validation  and  testing  of  the 
technology  are  unnecessary  requirements.”  This  is  an 
unacceptable  risk  because  every  piece  of  equipment 
must  meet  an  acceptable  set  of  requirements  relative  to 
the  DOD  operational  environment  for  which  it  is 
intended.  The  military  mission  and  operating 
environment  can  be  distinctly  different  than  those  of 
industry,  and  technologies  must  be  tested  and  validated 
to  withstand  factors  such  as  extreme  shock,  vibration, 
and  corrosion.  The  appropriate  level  of  testing  and 
validation  must  be  determined  based  on  the  type  of 
technology  and  how  it  will  be  fielded.  Ideally,  a COTS 
technology  may  be  subject  to  a reduced  level  of  DOD 
testing  based  on  established  commercial  testing  data.  If 
this  is  the  case,  the  DOD  will  realize  a cost  savings. 

When  specifying  COTS  technology,  parts 
obsolescence,  validation,  and  testing  risks  must  be 
effectively  balanced  with  system  performance,  life- 
cycle  costs  (affordability),  and  overall  supportability. 
One  management  tool  and  methodology  that  helps 
systems  engineers  identify  COTS  technology  risk 
factors  was  developed  by  the  Naval  Air  Warfare  Center 
Aircraft  Division,  Lakehurst,  New  Jersey 
(NAVAIRWARCENACDIVLKE)  under  the  Naval  Air 
Systems  Command.  The  Risk-Based  COTS  Systems 
Engineering  Assessment  Model  is  a tool  that  addresses 
the  need  for  better  systems  engineering  integrated 
decision-making.  The  model  can  improve  the  military 
systems  engineering  management  decision  framework 
so  that  COTS  technology  integration  is  considered  as 
an  alternative  to  “cradle-to-grave”  development  of 
DOD  weapon  systems.  Ultimately,  the  model  reduces 
risk  and  uncertainty  in  the  engineering  of  defense 
systems  that  use  COTS  technology. 

2.0  ACQUISITION  REFORM  INITIATIVE 

Several  key  measures  facilitate  the  accelerated 
introduction  of  commercial  technologies  into  DOD 
weapon  systems. 

2.1  COTS  Technology 

Various  DOD  directives  have  led  to  the  current  focus 
on  procuring  COTS  technology.  For  example,  DOD 
Directive  5000. 1 prescribes  a systems  engineering 
approach  throughout  the  entire  life  cycle  of  a system 
and  categorizes  the  four  basic  types  of  acquisition  in 
order  of  preference: 

a.  Modification  of  existing  system 

b.  Procurement  of  a COTS  item 

c.  Procurement  of  a nondevelopmental  item 

d.  Development  of  a new  system. 

The  DOD’s  Acquisition  Reform  Initiative  is  a 
mandated  effort  to  reduce  the  cost  of  systems 
acquisition  through  measures  such  as  COTS 
technology  procurement.  The  benefits  of  DOD 


acquisition  of  COTS  technology  can  be  significant, 
especially  with  respect  to  eliminating  developmental 
costs,  but  the  appropriate  risk  factors  must  be  explored 
for  each  unique  case. 

2.2  Open  System  Architecture 

A major  contributor  to  the  success  of  COTS-based 
technology  solutions  is  an  open  architecture  design. 
DOD  Directive  5000. 2-R  strongly  encourages  the 
design  of  open  architecture  for  DOD-developed 
systems  in  order  to  ensure  flexibility  and  scalability 
and  to  facilitate  the  insertion  and  integration  of 
technology.  In  many  cases,  industry  also  has  embraced 
open  architecture  in  order  to  promote  supportability, 
interoperability,  and  scalability  as  means  of  reducing 
production  costs  and  gaining  a competitive  advantage. 
Manufacturers  who  employ  the  principles  of  open 
architecture  represent  reduced  risk  to  the  DOD  when 
procuring  COTS  technology. 

Some  industry  standards  promote  open  architecture. 
For  example,  small  components  such  as  valves  often 
are  designed  using  open  architecture  standards  to 
ensure  that  they  can  be  applied  to  and  interchanged 
with  a wide  range  of  systems.  Unfortunately,  other 
types  of  mechanical  components,  such  as  pumps,  may 
not  be  adapted  as  easily  between  systems.  For  example, 
if  a manufacturer  develops  a system  that  includes  a 
unique  component  A,  which  for  some  reason  becomes 
unavailable  as  a replacement  part,  then  a new 
component  B,  possibly  from  a different  manufacturer, 
must  be  integrated.  If  the  system  was  not  designed  with 
open  standards  to  accommodate  a different  component, 
it  will  require  redesign  work  and/or  a new  interface  for 
component  B to  be  retrofitted  into  the  system. 

The  interchangeability  of  critical  parts  is  therefore  an 
important  factor  when  determining  the  risk  of  parts 
obsolescence  and  the  supportability  of  a COTS  system. 
The  COTS  systems,  which  are  designed  with  open 
architecture  and  open  standards,  yield  reduced  risk  and 
life-cycle  costs. 

3.0  SYSTEMS  ENGINEERING 

The  impetus  for  greater  application  of  COTS 
technology  creates  a new  systems  engineering 
challenge — to  cost-effectively  assess  and  integrate 
commercial  technologies  prone  to  continuous  change. 
Predicting  these  changes  and  ensuring  minimal  risk  can 
be  a difficult  task.  The  overall  goal  is  to  meet  mission 
requirements  while  ensuring  cost,  schedule,  and 
performance  throughout  the  weapon  system  life  cycle. 
This  goal  can  be  compromised  by  poorly  estimating  the 
risks  involved  with  COTS  technology  insertion. 

To  compensate  for  rapid  COTS  technology  changes, 
systems  engineers  must  identify  strategies  and  a 
common  framework  that  will  aid  in  projecting  and 
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mitigating  these  issues  early  in  the  weapon  system 
development  cycle.  By  addressing  market  (i.e., 
technology  manufacturer)  concerns  early,  the  volatility 
of  COTS  technology  insertion  can  be  controlled  and 
potential  problems,  such  as  parts  obsolescence,  can  be 
minimized. 

The  first  step  toward  meeting  this  objective  is  to  assess 
the  viability  of  the  commercial  technology  in  the 
context  of  performance,  complexity,  criticality, 
supportability,  and  life-cycle  cost  factors.  The  Risk- 
Based  COTS  Systems  Engineering  Assessment  Model 
is  a common  framework  that  allows  systems  engineers 
to  meet  the  goals  and  minimize  the  risks  of  COTS 
technology  insertion  at  any  phase  during  the  weapon 
system’s  acquisition  life  cycle.  The  model  can  be  used 
as  a life-cycle  risk  assessment  methodology  to 
determine  lifelong  buys  versus  COTS  technology 
insertion,  to  identify  open  architecture  and  open 
standards,  to  assess  supportability,  to  design  processes, 
and  to  select  materials.  It  is  a life-cycle  management 
tool  for  dealing  with  the  risk  of  obsolescence  and 
overcoming  the  barriers  to  using  COTS  technology  in 
defense  systems.  An  innovative  aspect  of  the  model  is 
the  use  of  a cube  diagram  to  represent  the  relative  risks 
of  different  COTS  alternatives  (reference  Section  4.3). 

4.0  ASSESSMENT  AND  VALIDATION 
MODEL 

The  Risk-Based  COTS  Systems  Engineering 
Assessment  Model  was  developed  to  ensure  that 
systems  engineers  can  select  the  most  cost-effective 
COTS  equipment  based  on  its  affordability,  reliability, 
mission  requirements,  and  ability  to  accommodate 
replacement  and/or  future  modification.  A risk-based 
approach  to  decision-making,  the  model  enables 
systems  engineers  to  apply  a variety  of  risk 
perspectives  while  using  information  from  technology 
market  analyses.  For  example,  market  analysis 
information  can  be  used  to  assess  whether  a 
manufacturer  uses  open  architecture  or  is  likely  to  have 
the  “staying  power”  to  provide  long-term  support.  The 
model  also  assists  with  determining  the  level  of 
validation  and  testing  required  to  further  reduce  the 
risk  of  using  COTS  equipment.  The  model  allows 
competing  COTS  equipment  to  be  judged  fairly  in 
order  to  identify  which  manufacturer  allows  the  DOD 
to  take  the  greatest  advantage  of  using  COTS 
equipment  (e.g.,  the  manufacturer  whose  technology 
meets  the  mission  requirements,  uses  open  architecture, 
and  provides  verifiable  data  to  limit  the  amount  of 
DOD  testing  and  validation  required). 

For  instance,  the  model  can  assist  in  recognizing  the 
worst-case  parts  obsolescence  scenario — selecting 
equipment  that  has  a high  perceived  risk  of  not 
functioning  during  a conflict  as  a result  of  the 
unavailability  of  parts  or  the  incompatibility  of  newly 
upgraded  parts  with  fielded  equipment.  The  best-case 


scenario  is  one  in  which  the  equipment  meets  the 
mission  by  using  readily  available  and  supportable 
COTS  parts  and  open  architecture.  In  this  case, 
components  can  be  replaced  to  compensate  for,  as  well 
as  to  take  advantage  of,  advances  in  technology. 

The  model  is  intended  to  be  a tool  that  can  be  applied 
throughout  the  lifetime  of  a system.  Ideally,  the  model 
should  be  used  to  perform  a baseline  analysis  when 
system  development  commences.  The  analysis  can  be 
revised  and  adjusted  later  during  each  major  milestone 
or  acquisition  phase  to  account  for  new  requirements  or 
factors  that  were  not  originally  relevant  or  defined.  If  a 
weapon  system  has  progressed  beyond  the 
development  stage,  the  model  can  still  be  applied  at 
any  time  to  assist  with  COTS  technology  decision- 
making. The  model  functions  best  when  it  is  combined 
with  a suitable  life-cycle  cost  model. 

Overall,  the  model  works  iteratively  to  define 
requirements,  insert  market  knowledge,  and  identify 
risk.  Each  COTS  alternative  is  applied  to  the  model.  If 
alternative  1 yields  unacceptable  risk,  consecutive 
alternatives  are  evaluated  until  the  altemative(s)  with 
the  least  risk  is  identified.  If  all  of  the  available 
alternatives  have  unacceptable  risk,  either  the  mission 
requirements  must  be  reevaluated  or  other  suitable 
alternatives  must  be  found  through  additional  market 
analysis.  The  model  defines  risk  as  a function  of 
mission  criticality,  technical  complexity,  and  life-cycle 
costs.  For  example,  the  risk  of  parts  obsolescence  is 
translated  as  a risk  to  the  mission  and  as  a potential 
impact  on  life-cycle  costs.  Furthermore,  unless  the  item 
has  been  designed  using  open  architecture,  the  risk  of 
parts  obsolescence  is  evaluated  according  to  the 
technical  complexity  of  the  COTS  technology — the 
more  technically  complex  the  technology,  the  greater 
the  perceived  risk  of  parts  obsolescence. 

The  goal  of  engineering  suitable  COTS  equipment 
solutions  can  be  reached  by  employing  the  model  in 
accordance  with  the  following  steps: 

a.  Perform  market  surveillance  and  construct  an 
ongoing  commodity  strategy  for  future  needs. 

b.  Logical  Solution — Perform  an  operational 

requirements  analysis  (e.g.,  define  mission, 
performance,  functionality,  reliability, 

maintainability,  supportability,  and  environmental 
requirements). 

c.  Physical  Solution — Translate  requirements  into 
COTS  solutions  by  applying  market  analysis. 

d.  Alternatives  Risk  Assessment  (a  central  element 
of  the  model) — Perform  an  alternatives  and  risk 
assessment 
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> Evaluate  the  ability  of  each  alternative  to  meet 
the  defined  requirements. 

> Determine  the  requirements  thresholds. 

> Determine  the  requirements  validation  and 
testing  required. 

> Determine  supportability  plans  and  evaluate 
open  architecture  design. 

> Determine  risk  factors  to  performance,  cost, 
and  schedule. 

> Determine  the  estimated  life-cycle  cost. 


e.  Mitigation  of  Risk — Perform  verification  and 

qualification 

> Analyze  commercial  data  and  past 
performance 

> Determine  required  testing  and  validation 
of  sample  equipment. 

Figure  1 summarizes  these  steps  and  shows  how  the 
model  fits  into  the  traditional  systems  requirements 
decision-making  process. 


Figure  1.  Summary 


Figure  2 represents  the  iterative  process  that  occurs 
after  the  need  for  a piece  of  equipment  is  defined. 
Blocks  1,  2,  and  3 relate  to  defining  requirements, 
determining  market-based  COTS  solutions,  and 
assessing  each  COTS  alternative.  If  none  of  the  COTS 
alternatives  represents  acceptable  risk,  the  mission 
requirements  must  be  reevaluated  or  a decision  must  be 


made  to  develop  the  equipment  in-house  (DOD  design 
and  develop)  rather  than  procuring  COTS  equipment. 
If  one  or  more  COTS  alternatives  represent  acceptable 
risk,  a procurement  strategy  for  COTS  equipment 
should  be  formulated  based  on  the  best  alternative. 
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Figure  2.  Iterative  Decision  Analysis  Process 


4.1  The  Logical  Solution 

Block  1 of  Figure  2,  Perform  Operational 

Requirements  Analysis,  includes  the  following 

substeps: 

• Compile  user  input  (e.g.,  feedback  from 
shipboard/ flight  line  personnel). 

• Define  the  mission  profile  and  mission  analysis. 

• Define  thresholds  and  objectives. 

• Perform  a functional  analysis. 

• Perform  a supportability  analysis. 

• Define  performance  attributes. 

• Determine  operational  availability  (Ao),  allowable 
mean  time  between  failures  (MTBF),  and  mean 
time  to  repair  (MTTR). 

• Define  the  operational  environment  requirements 
(e.g.,  shock,  vibration,  weather). 

• Determine  estimated  inventory  and  allocation 

allowances. 

4.2  The  Physical  Solution 

Block  2 of  Figure  2,  entitled  Translate  Requirements 

into  COTS  Solutions,  includes  substeps  such  as 


performing  market  research,  analyzing  market  data, 
and  surveying  COTS  equipment  suppliers.  Market 
research  builds  on  continuous  market  surveillance  to 
develop  a commodity  strategy  and  market 
investigation.  The  market  investigation  should  yield 
COTS  alternatives  that  meet  the  requirements  of  the 
logical  solution  (defined  above).  A typical  market 
investigation  results  in  an  evaluation  and  report  of  the 
following  items: 

• Summary  of  market  surveillance  infonnation 

• List  of  potential  sources 

• Survey  of  potential  supply  sources  (e.g.,  Internet 
search,  journals,  Commerce  Business  Daily 
contract  awards,  etc.) 

• Input  from  references  (i.e.,  current  users  of  similar 
equipment) 

• Compilation  of  equipment  capabilities  (e.g., 
performance,  supportability,  history,  etc.). 

Table  1 lists  some  factors  that  should  be  considered 
when  reviewing  open  standards,  equipment  profiles, 
and  their  related  technologies  and  products. 
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Table  1.  Market  and  Technology  Supplier  Analysis 

(Source:  Next  Generation  Computer  Resources  (NGCR),  Document  No.  AST002,  Version  0. 04  of  the  NGCR 
Support  ability  Guide,  draft  dated  27  April  1995,  SPAWAR.) 


Maturity  of  the  Standards, 
Technologies,  and  Products 

> Is  the  technology  mature? 

> Are  the  products  fairly  stable? 

> What  is  the  product  “upgrade”  cycle  time? 

> When  is  the  next  planned  update? 

> Are  the  products  being  refined  or  significantly  changed  during  each  cycle? 

Multiple  Product  Sources 

> Are  there  multiple  sources  for  products  that  meet  the  requirements  analysis? 

> Are  these  products  interoperable? 

> Do  these  products  merely  accept  data  from  each  other  or  do  they  meet  the  same 
performance  levels  (interchangeability)? 

Market  Acceptance 

> Is  the  standard,  profile,  or  product  well  accepted  in  the  commercial  marketplace? 

> What  are  the  respective  vendors’  market  shares? 

r Are  the  commercial  markets  large  enough  to  imply  that  long-term  support  and 
upgrade  of  the  product  will  be  an  investment  borne  by  the  commercial  market 
sector  or  will  the  DOD  become  the  only  user  in  a relatively  short  time? 

Product  Line  Families 

r Do  product  families  exist? 

> Will  usage  of  a given  product  tie  the  DOD  to  a product  family? 

> Will  such  a relationship  be  expensive? 

> Is  the  existing  support  structure  well-suited  to  the  operational  requirements? 

> Will  supplements,  upgrades,  or  replacements  be  necessary  (e.g.,  technical  data, 
training,  repair,  spare  parts  support,  etc.)? 

> Should  the  product  family  or  the  individual  product  alone  be  approved  for  use? 

Test  and  Evaluation 

> What  ongoing  test  and  evaluation  parameters  are  employed  by  the  vendor? 

> How  would  the  DOD  test  this  product? 

> Will  the  existing  test  capability  and  data  meet  the  DOD’s  needs? 

> Will  test  data  from  families  of  products  be  applicable? 

> How  much  will  required  testing  cost? 

Technical  Data 

> Are  the  technical  data  provided  by  the  various  vendors  sufficient? 

> Are  the  data  useable?  If  no,  what  problems  can  be  foreseen? 

> What  workarounds  are  necessary? 

> What  additional  data  are  necessary? 

Configuration  Management 
(CM) 

> Is  the  contractor’s  CM  program  adequate  to  meet  weapon  system  program  office 
needs? 

> Can  the  contractor’s  CM  program  be  modified  or  supplemented  if  necessary? 

By  the  contractor  or  the  government? 

> What  will  the  cost  be  and  who  will  bear  this  cost? 

Availability 

> What  is  the  operational  availability  (Ao)? 

> What  is  the  inherent  availability? 

> What  is  the  mean  time  to  repair  (MTTR)? 

> What  is  the  mean  time  between  failures  (MTBF)? 

Performance  Monitoring 
and  Built-in  Test 

> Does  the  product  have  a built-in  self-test? 

> Is  the  self-test  capability  sufficient  from  a systems-level  viewpoint? 

> Will  the  self-test  be  difficult  to  reintegrate  when  updates  occur  (e.g., 
engineering,  training,  configuration  status  and  management,  supply  support)? 

Quality  Assurance 

> Does  the  vendor  provide  a warranty  and  what  is  included  in  the  warranty? 

> Is  the  vendor  ISO  9000  compliant? 

> What  other  quality  assurance  measures  does  the  vendor  provide? 
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4.3  The  Alternatives  Risk  Assessment 

Block  3 of  Figure  2,  entitled  Perform  COTS 
Assessment,  includes  the  following  substeps: 

• Classify  each  COTS  alternative  based  on  criticality 
and  complexity.  (Since  each  alternative  is  a 
possible  solution  for  the  same  need,  it  is  expected 
that  the  criticality  will  remain  the  same  for  each 
alternative;  however,  the  complexity  may  vary 
with  each  alternative.). 

• Evaluate  the  anticipated  life-cycle  cost  analysis  for 
each  COTS  alternative. 

• Assess  each  COTS  alternative  based  on: 

> Ability  to  meet  threshold  and  objective 
requirements 

> Supportability  (e.g.,  open  architecture  design 
reduces  parts  obsolescence) 

> Life-cycle  cost. 

• Assess  the  risk  of  each  COTS  alternative: 

> Technical  risk  = / (mission  criticality, 
technical  complexity,  life-cycle  cost  [LCC]) 

To  perform  the  alternatives  risk  assessment — the 
central  element  of  the  model — the  technical  complexity 
and  criticality  of  each  COTS  alternative  must  be 
established.  The  alternatives  are  categorized  using  the 
following  definitions: 


• Complexity 

> Non-Complex  - A nonrepairable  piece  of 
equipment  (i.e.,  consumable)  or  a repairable 
piece  of  equipment  with  no  repairable 
subassemblies. 

> Complex  I - Equipment  with  one  or  more 
repairable  subassembly. 

> Complex  II  - Equipment  that  meets  the 
definition  of  Complex  I and  is  self-powered 
(i.e.,  engine,  hydraulic,  electric,  or  pneumatic- 
powered). 

> Complex  III  - Equipment  that  meets  the 
definition  of  Complex  II  and  has  feedback 
control  (i.e.,  does  not  have  data  acquisition). 

• Criticality 

> Non-Critical  - Requires  scheduled  and/or 
unscheduled  maintenance,  but  is  not 
considered  mission-  or  safety-critical. 

> Mission  Critical  — Failure  of  this  equipment 
could  damage  the  weapon  system  or  degrade 
the  weapon  system  mission. 

> Safety  Critical  - Failure  of  this  equipment 
could  harm  personnel. 

Next,  the  equipment  alternatives  should  be  assessed  to 
determine  approximate  life-cycle  costs.  At  this  point, 
the  alternatives  can  be  positioned  on  a three- 
dimensional  cube  that  forms  the  basis  of  the  Risk- 
Based  COTS  Systems  Engineering  Assessment  Model 
(refer  to  Figure  3). 
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Figure  3.  Degree  of  Validation  as  a Function  of  Technical  Risk 

Risk  ^/(mission  criticality,  technical  complexity,  LCC) 


This  cube  allows  the  systems  engineer  to  determine  the 
degree  of  validation  required  as  a function  of  technical 
risk.  Risk  is  a function  of  three  factors:  criticality, 

complexity,  and  life-cycle  cost.  The  cube  enables 


systems  engineers  to  visualize  alternatives  as  a 
composite  of  their  contribution  to  the  mission  versus 
their  ease  of  repair  and  supportability  versus  cost.  The 
y-axis  of  the  cube  represents  increasing  complexity, 
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and  the  x-axis  represents  increasing  criticality.  The  z- 
axis  represents  increasing  life-cycle  costs.  Each 
available  COTS  alternative  should  be  positioned  in  a 
sector  of  the  cube.  The  cube  is  color-coded  to  indicate 
which  sectors  represent  low,  moderate  and  high  risk, 
which  correspond  to  low,  moderate,  and  high 
requirements  for  equipment  validation.  For  example, 
the  color-coded  location  of  the  sector  for  those 
alternatives  that  are  noncritical  and  noncomplex  with  a 
low  life-cycle  cost  indicates  low  risk  and,  therefore, 
relatively  low  requirements  for  equipment  validation. 
The  color-coded  location  of  the  sector  for  those 
alternatives  that  are  highly  mission-  or  safety-critical 
and  highly  complex  with  a high  life-cycle  cost 
indicates  high  risk  and  relatively  high  validation 


requirements.  The  model  also  indicates  potential 
acquisition  problems,  such  as  alternatives  that  fall  into 
the  sector  for  low  complexity  and  low  criticality  with 
high  life-cycle  costs.  Such  sectors  are  color-coded  to 
indicate  either  a problem  with  the  availability  of  an 
appropriate  alternative  in  the  marketplace  or  that  the 
requirements  have  been  poorly  defined. 

Figure  4 illustrates  a fragmented  version  of  the  cube 
that  enables  better  visualization  of  each  sector.  This 
model  expands  the  cube  to  include  sectors  based  on  all 
four  definitions  of  complexity. 
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Figure  4.  Degree  of  Validation  as  a Function  of  Technical  Risk  - Fragmented  Cube 

Risk  =/ (mission  criticality,  technical  complexity,  LCC) 


As  an  example  of  the  different  components  and  support 
equipment  that  comprise  a weapon  system.  Figure  5 is 
a version  of  the  fragmented  cube  with  various  pieces  of 
aircraft  support  equipment  labeled  on  the  appropriate 
sectors.  By  using  the  fragmented  cube  to  visualize  an 


entire  weapon  system,  systems  engineers  can  select  the 
areas  that  may  be  most  appropriate  for  COTS 
equipment  to  be  inserted  and/or  ensure  that  the 
appropriate  level  of  validation  occurs  when  evaluating 
COTS  equipment  based  on  risk. 
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EQUIPMENT 

SAMPLES 


Acronym  List 

SETS-standard  engine  test  system 
HCTS-hydraulic  component  test  stand 
JASU-jet  air  start  unit 
ADTS-air  data  test  set 
MEPP-mobile  electric  power  plant 
AGTS-aircraft  generator  test  stand 
NDI-nondestructive  inspection  equipment 
02  GEN-oxygen  generating  cart 
MAINT  PLAT-maintenance  platform 


COMPLEX  111 


COMPLEX  II 


COMPLEX  I 


NON-COMPLEX 


NON  MISSION/ 

CRITICAL  SAFETY 

CRITICAL 

► Higher Ao  Required 


Figure  5.  Weapon  System  Example  of  Fragmented  Cube 

Risk  =/ (mission  criticality,  technical  complexity,  LCC 


4.4  The  Mitigation  of  Risk 

When  assessing  COTS  alternatives,  it  is  necessary  to 
determine  what,  if  any,  performance  and  environmental 
degree  of  reliability  and  maintainability  (R&M) 


validation  and  testing  are  required.  Figure  6 shows  an 
example  of  a logical  flow  diagram  validation  strategy 
for  COTS  equipment  R&M  validation  decision  factors 
based  on  criticality  and  complexity.  The  goal  is  not  to 
“overtest”  or  “undertest”  COTS  equipment. 


Figure  6.  Reliability  and  Maintainability  Validation  Strategies 

(Source:  Janet  L.  French,  NA  VAIR  Reliability  Engineering) 


Tables  2 and  3 represent  the  types  of  commercial 
equivalent  data  and  equivalent  testing  that  can  be  used 
to  assess  the  degree  of  additional  testing  or  validation 
that  may  be  required.  In  each  case,  a lack  of 
commercial  data  or  testing  protocols  increases  risk 


and  may  necessitate  full  or  partial  DOD  testing  of  the 
equipment.  The  goal  is  to  take  advantage  of  existing 
data  and  testing  to  reduce  the  cost  of  required  R&M 
testing  and  validation. 
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Table  2.  R&M  Validation 
Analysis  and  Data 


Analysis  Data 
Required 

Critical  & 
Noncomplex 

Complex 
& Not 
Critical 

Electrical/ 
Electronic 
Critical  & 
Complex 

Mechanical 
Critical  & 
Complex 

Critical  & 
Complex  II 

Critical  & 
Complex  III 

R design  practices 

/ 

/ 

/ 

R prediction 

/ 

/ 

/ 

FMECA 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

M prediction* 

/ 

/ 

/ 

*Maintenance  philosophy-dependent 


Table  3.  R&M  Validation 
Testing 


Testing 

Critical  & 
Noncomplex 

Complex  & 
Not  Critical 

Electrical/ 
Electronic 
Critical  & 
Complex 

Mechanical 
Critical  & 
Complex 

Critical  & 
Complex  II 

Critical  & 
Complex  III 

/ 

/ 

y 

RQT 

/ 

/ 

y 

y 

RD/GT* 

M demo** 

/ 

/ 

y 

y 

*For  systems  where  several  COTS  items  are  integrated. 
**Maintenance  philosophy  dependent 


Legend: 

R — reliability 

FMECA — failure  modes  effects  and  criticality  analysis 

M — maintainability 

ESS — environmental  stress  screening 

RQT — reliability  qualification  testing 

RD/GT — reliability  development/growth  testing 


When  conducting  an  R&M  risk  assessment,  the 

following  pertinent  questions  should  be  included: 

• Flas  the  vendor  provided  sufficient  information  to 
indicate  that  R&M  requirements  can  be  achieved? 

• Are  there  any  new  or  untried  technologies  or 
components  within  the  product  that  have  a limited 
or  nonexistent  record  of  reliability  performance? 

• What  techniques  does  the  vendor  use  to  maintain 
or  improve  product  reliability  and  quality? 

• Flow  does  the  vendor  select  subvendors  (e.g., 
qualified  lists,  lowest  cost,  etc.)? 

• Does  the  vendor  verify  component  quality? 


• Are  there  any  frequent  failures  that  could  impact 
safety  or  the  mission? 

• Are  there  any  frequent  failures  of  high-cost  items? 
Hard-to-replace  items?  Hard-to-maintain  items? 

• Is  the  commercial  use  environment  sufficiently 
similar  that  the  data  are  indicative  of  the  types  of 
failures  likely  in  the  DOD  environment? 

• Are  there  any  test  data  and  are  they  verifiable? 

Figure  7 illustrates  the  decision  factors  related  to 
COTS  equipment  supportability  validation 
requirements.  This  analysis  shows  that  open 
architecture  is  beneficial  to  COTS  equipment 
alternatives. 
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Figure  7.  Supportability  Validation  Strategies 
Logistics  Validation 

(Source:  Edward  F.  Waraksa,  NA  VAIR  Logistics  Management) 


Various  types  of  commercial  data  may  be  available  to 
perform  R&M  and  supportability  analyses  of  COTS 
alternatives.  Table  4 illustrates  several  key  sources. 


Similar  validation  flow  diagram  strategies  must  be 
developed  to  address  performance  as  well  as 
environmental  requirements. 


Table  4.  Commercial  Data  Sources  Related  to  Validation  of  R&M  and  Supportability 


Historical  R&M  Experience 

• Estimates  of  expected  reliability 

• Warranty  provisions 

• Customer  satisfaction  indices 

Internal  Manufacturing  Quality 
Procedures 

• Production  controls 

• ISO  9000  or  similar  techniques 

• Testing  procedures 

Vendor  or  Component  Selection  Policy 

• Parts  control  methods 

• Quality  control  techniques 

• Testing  procedures 

• Environmental  stress  screening 

Design  Approach 

• Environmental  approach 

• Part  derating  procedures 

• Fault  tolerance  features 

• Ruggedization  concepts 

• Built-in  test  features 

• Ease-of-maintenance  features 
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Iii  summary  (refer  to  Figure  1),  the  Risk-Based  COTS 
Systems  Engineering  Assessment  Model  components 
can  be  summarized  as  the  following  steps  - beginning 
with  the  mission  need,  requirements  definition  and 
analysis,  market  research  and  identification  of  COTS 
solutions,  use  of  the  fragmented  cube  and  validation 
flow  charts  to  assess  and  reduce  risk  and,  finally, 
providing  input  for  procurement. 

5.0  CONCLUSIONS 

In  summary,  the  DOD’s  increasing  reliance  on 
advanced  technology,  such  as  electronics,  dramatically 
increases  the  cost  of  developing  weapon  systems,  as 
well  as  the  operational  cost  of  redesigning  and 
upgrading  these  systems  as  technologies  change.  To 
avoid  some  of  these  costs,  the  DOD  must  take 
advantage  of  industry’s  ability  to  bring  components 
and  systems  to  market  faster  than  the  DOD  can  develop 
them.  It  is  important  to  weigh  the  risks  of  COTS 
technology  over  the  life  cycle  of  the  system  and  insert 
these  commercial  technologies  where  the  risks  and 
benefits  are  prudent.  Without  proper  management, 
COTS  can  be  a drawback  as  a result  of  poorly  defined 
risk — such  as  the  likelihood  of  performance  and  cost 
risks  as  well  as  parts  obsolescence  in  the  field.  The 
Risk-Based  COTS  Systems  Engineering  Assessment 
Model  serves  to  define  such  risks  and  helps  systems 
engineers  make  informed  decisions. 

The  Risk-Based  COTS  Systems  Engineering 
Assessment  Model  provides  a common  framework  for 
making  COTS  technology  decisions  by  assessing  the 
relative  risk  of  each  COTS  alternative.  It  also  provides 
assistance  in  determining  the  appropriate  degree  of 
validation  required  to  verify  that  a COTS  alternative 
can  be  transferred  to  the  military  environment. 

To  take  advantage  of  COTS  technology  and  better 
apply  the  model,  the  systems  engineering  community 
needs  the  following: 

• Better  requirements  analysis  tools  that  incorporate 
risk. 

• Better  industry  information  (i.e.,  life-cycle  cost 
data,  time  until  market  release/update  data,  and 
supportability  data. 

• Better  market  surveillance  and  segmentation  (i.e., 
systems  engineers  must  become  cognizant  of 
market  factors  and  sectors  for  different 
technologies). 

• Better  system  of  open  architecture  standards  in  the 
marketplace  (e.g.,  electronic  and  mechanical 
standards  that  incorporate  open  architecture). 

• Better  assessment  tools  that  are  standardized  and 
used  by  all  NATO  military  organizations. 

The  Risk-Based  COTS  Systems  Engineering 
Assessment  Model  offers  important  benefits  and 


insight  to  the  overall  weapon  system  acquisition 
management  process.  It  should  be  noted  that 
significant  work  must  still  be  invested  to  make  the 
application  more  efficient,  such  as  refining  the 
functional  interrelationship  between  complexity, 
mission,  and  cost.  The  need  to  further  optimize  the 
model  is  necessary  if  better  fidelity  is  desired. 
Integrating  and  automating  the  complexity  criteria  with 
mission  criticality  and  cost  analysis  is  the  ideal  formula 
for  concurrent  engineering  analysis,  which  when 
applied  improves  the  chances  of  selecting  effective 
commercial  equipment.  Automating  and  combining 
the  model  can  significantly  improve  implementation 
and  accelerate  the  transfer  of  commercial  technology  in 
a synergistic  manner. 

The  authors  would  like  to  acknowledge  Janet  L.  French 
and  Edward  F.  Waraksa  for  their  contributions  to  this 
paper. 
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